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(54) Video transmission rate matching for multimedia communication systems 



(57) Digital video transmission rate matching tech- 
niques are disclosed. A bit stream rate matching appa- 
ratus includes a bit rate increasing device and a bit rate 
reducing device (107). The bit rate increasing device 
converts a video bit stream having a first transmission 
rate to a video bit stream having a second transmission 
rate characterized in that the first transmission rate is 
less than the second transmission rate. The bit rate re- 



ducing device (107) converts a video bit stream (101) 
having the second transmission rate to a video bit 
stream having the first transmission rate. The bit stream 
rate matching apparatus is useful in the context of a mul- 
timedia conference where a first endpoint device em- 
ploys the first transmission rate and a second endpoint 
device employs the second transmission rate. 
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Description 

Background of the Invention 

1. Field of the Invention 

The invention relates generally to multimedia com- 
munication systems which utilize endpoint devices, and 
more specifically to techniques for matching the video 
transmission rates of a plurality of endpoint devices. 

2. Descnptionof the Prior Art 

Video transmission rate matching is a technique 
which has been used to convert the bit rate of a first 
video bit stream to match that of a second video bit 
stream This conversion, may require bit rate reduction 
and/or bit rate increases. Among these two conversion 
processes bit rate reduction is more critical, due to the 
fact that thts process involves the removal of bits. Since 
these bits represent video information, the quality of a 
video signal having a reduced bit rate may be degraded. 
The amount of degradation is related to the manner in 
which the bit rate reduction is achieved. 

With respect to bit rate increases, no information is 
removed from the video bit stream Instead, such a rate 
increase requires the incorporation of additional bits into 
the bit stream Although it would be desirable to add bits 
to the bit stream for the purpose of enhancing the video 
signal, such enhancements are often in the pel domain. 
For example if an incoming video signal has been sub- 
jected to a process of data compression, the information 
discarded in the compression process is not stored. The 
information is lost, and cannot be recovered and added 
to the bit stream during the process of increasing the bit 
rate. Therefore, bit rate increases are implemented by 
adding "extra" bits to the bit stream, wherein these "ex- 
tra" bits are not the same bits that may have been re- 
moved during a previous data compression step. Bits 
are added using conventional bit- stuffing patterns tooc- 
cupy the additional bandwidth of a rate-increased video 
signal. 

Video transmission rate matching has been applied 
to a wide variety of applications in the field of multimedia 
communication, especially in environments involving 
multipoint communication topologies. For example, in 
multimedia conferencing applications, a "bridge" or 
"multipoint control unit" (fvlCU) is typically employed. 
This MCU is a computer-controlled device which in- 
cludes a multiplicity of communication ports. The ports 
may be selectively interconnected in a variety of ways 
to provide communication among a group of endpoint 
devices. MCUs are often used to establish multi-party 
conferences among a plurality of endpoint devices, and 
to establish multi-party conference calls. A state-of-the- 
art MCU is described in ITU Document H 243. "Proce- 
dures for Establishing Communication Between Three 
or More Audiovisual Terminals Using Digital Channels 



up to 2 Mbps". March 1993. and in ITU Document H. 
231. "Multipoint Control Units for Audiovisual systems 
Using Digital Channels up to 2 Mbps". March 1993. 
Existing MCUs require all endpoint devices partici- 

s pating in a given multimedia conference to use the same 
video transmission rate. Typically, during the initial set- 
ting up of a multimedia conference, or at the time that it 
is desired to add an additional endpoint device to an ex- 
isting conference, the MCU polls ail of the endpoint de- 

to vices to ascertain the video transmission rate or rates 
each endpoint device is equipped to implement. When 
the MCU polls a given endpoint device, the endpoint de- 
vice transmits a capability code back to the MCU. The 
capability code includes information specifying the vid- 

>5 eo-bit rate transmission capabilities, audio bit rate trans- 
mission capabilities, etc., of a given endpoint device. 
These capability codes may also specify the compres- 
sion algorithm or algorithms used by a given endpoint 
device. 

20 Based upon the capability codes received by the 
MCU from the various endpoint devices, the MCU se- 
lects a minimum video bit rate to be used as the video 
transfer rate for the multimedia conference. The MCU 
sends a video rate signal to all endpoint devices which 

25 specifies use of this video transfer rate. Upon receipt of 
the video rate signal, the endpoint devices all operate 
using the same video transfer rate. 

Use of the same video transfer rate for all endpoint 
devices participating in a given conference presents sig- 

30 nificanl shortcomings. For example, in many cases, a 
conference participant may desire to receive high-qual- 
ity video by using a relatively high video transfer rate, 
whereas other participants may want to use less expen- 
sive equipment offering a lower video transfer rate. 

35 State-of-the-art MCUs cannot implement a multimedia 
conference using a plurality of different video transfer 
rates. Therefore, it would be desirable to equip existing 
MCUs with a video transmission rate matching device. 
Through the use of video transmission rate matching 

-w techniques, video communication among a plurality of 
endpoint devices will be provided, even when the end- 
point devices use different video transmission rates. 

One video transmission rate matching method well- 
known to those skilled in the art is termed the transcod- 

-*5 ing method. Pursuant to the transcoding method, a com- 
pressed video bit stream having a first bit rate is fully 
decoded into a video space known as the pel domain. 
This fully-decoded bit stream, which may be conceptu- 
alized as a completely reconstructed video sequence. 

50 is then encoded into a video bit stream having a second 
bit rate. 

The existing transcoding method is disadvanta- 
geous Since decoding as well as encoding processes 
are required, transcoding is very time-consuming. As a 
55 practical matter, the time delay is at least twice that of 
the end-to-end encoding delay Such a delay is not tol- 
erable for applications requiring real-time communica- 
tion, such as multimedia conferencing A (aster method 
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of performing video bit rale matching is needed 

Summary of the Invention 

Digital video transmission rate matching techniques 
are disclosed. A bit stream rate matching apparatus in- 
cludes a bit rate increasing device, a bit rate reducing 
device, and first and second switching devices. The first 
switching device switches each of a plurality of incoming 
video bit streams to any one of the bit rale increasing 
device, the bit rate reducing device, and the second 
switching device. Each of these incoming video bit 
streams originates from a particular endpoint device 
The second switching device switches video bit streams 
from the bit rate reducing device, the bit rate increasing 
device, and the first switching device to each of a plu- 
rality of outgoing video bit streams. The first and second 
switching devices are controlled by a processor. The bit 
stream rate matching apparatus is useful in the context 
of a video communication where a first endpoint device 
employs the first transmission rate and a second end- 
point device employs the second transmission rate. 

Brief Description of the Drawings 

FIG. 1 is a hardware block diagram of a video trans- 
mission rate reduction apparatus according to a 
preferred embodiment disclosed herein: 
FIGs. 2 and 3 are data structure diagrams setting 
forth illustrative coding formats for representing 
video information. 

FIG. 4 is a data structure diagram illustrating the 
arrangement of DCT coefficients within one of the 
blocks shown in FIG. 3: 

FIG. 5 is a data structure diagram setting forth an 
illustrative example of DCT coefficient partitioning: 
FIG. 5 is a software flowchart setting forth a proce- 
dure for performing video bit rate matching accord- 
ing to a preferred embodiment disclosed herein: 
FIG. 7 is a hardware block diagram setting forth an 
illustrative structure for the discrete cosine transfor- 
mation (DCT) processor of FIG. 1: 
FIG. 6 is a hardware block diagram setting forth the 
system architecture of a multimedia conferencing 
system which includes an MCU having a video proc- 
essor equipped to perform video transmission rate 
matching methods according to various preferred 
embodiments disclosed herein: 
FIG 9 is a block diagram showing an illustrative 
hardware configuration for the video processor of 
FIG 6; and 

FIG. 10 is a hardware block diagram setting forth a 
bit rale matching system for use with five endpoint 
devices. 

Detailed Description 

For illustrative purposes video transmission rate 



matching wilt be described in the operational context of 
an H 261 environment. However it is to be understood 
that the video transmission rate matching techniques 
described herein are generally applicable to any video 

5 compression algorithm which uses transformation and 
quantization processes. For example the techniques 
disclosed herein may be employed in conjunction with 
video telephones of the type described in CCITT recom- 
mendation COST 211. However, for purposes of the 

to present disclosure, video data to and/or from the video 
transmission rate matching system will be compressed 
in an H.261 -like format. 

An "H 261 -like" format is any coding format which 
is similar to the coding format currently being estab- 

*5 lished by the international Telecommunications Union 
(ITU-T). The format is described in the ITU-T document 
"Recommendation H.261. Video Codec for Audiovisual 
Services at px54 kbits/sV May 1992. and the ITU-T doc- 
ument "Description of Reference Model 5". June 9. 

20 1989. 

FIG. 1 is a hardware block diagram showing a trans- 
mission rate reduction system 100 for a video signal 
which has been compressed in an H.25i-like format. 
The transmission rate reduction system 100 includes an 

25 input port 10 1 for accepting a compressed video signal 
having a first bit transmission rate and an output port 
116 for providing a compressed video signal having a 
second bit transmission rate. A compressed video sig- 
nal is defined as the binary representation of a video 
signal which has been compressed by a coding algo- 
rithm substantially similar to that described in the H.261 
standard, and then coded according to a syntax sub- 
stantially similar to that described in the H.261 standard. 
A compressed video signal m the form of a digital. 

35 coded bit stream is presented to input port 101. This 
compressed video signal includes coded digitized video 
information along with a header The header may in- 
clude the types of information specified in the H.261 
standard = and/or other types of information such as the 

•to source, the destination, the content, and/or the organi- 
zation of the video information 

Input port 101 is connected to a receiving buffer 
1 02. which is a conventional digital buffer. This receiving 
buffer 102 provides electrical isolation between the 

4 5 source of the compressed video signal and the various 
circuit elements shown in FIG 1 

The output of the receiving buffer 102 is coupled to 
a video multiplex decoder (VMD) 104. VMD 104 in- 
cludes the combination of a decoder and a demulliplex- 

50 er. The demultiplexer is equipped to demultiplex the 
coded bit stream. The decoder decodes header infor- 
mation which has been coded into the aforementioned 
coded bit stream. The demultiplexer and decoder per- 
form the function of recovering compressed video data 

55 from the coded bit stream. 

The VMD 104 includes a first output port 1 05 and a 
second output port 106. The VMD 104 provides the first 
output port 105 with quantized DCT coefficients along 
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with quantization information. The nature of these DCT 
coefficients is described in greater detail in the H.261 
standard. The second output port 106 is provided with 
motion vectors. First output port 105 ts coupled to DCT 
coefficients processor 107. and second output port 106 
is coupled to a video multiplex encoder (VME) 1 09. The 
operation and structure of VME 109 will be described in 
greater detail below. The motion vectors are sent direct- 
ly from VMD 104 to VME 109 because no motion esti- 
mation is employed in the present example. 

The DCT coefficients received from the first output 
105 of VMD 104 are sent to DCT coefficients processor 
107 The DCT coefficients processor 107 processes the 
DCT coefficients in such a manner that the output 106 
of DCT processor 107 is a signal which requires fewer 
bits to encode than the number of bits which were re- 
ceived from the first output 105 of the VMD 104. The 
DCT coefficients processor 107 reduces the number of 
bits such that the resulting video quality is not substan- 
tially degraded as perceived by human visual process- 
es. In other words the video quality is degraded "grace- 
fully". The amount of information reduced by processing 
coefficients at DCT coefficients processor 107 is con- 
trolled by the rate-control signal 1 14 sent by rate control 
unit 113. Various techniques for programming DCT co- 
efficients processor 107 to degrade video quality grace- 
fully are well known to those skilled in the art. Three such 
techniques for gracefully degrading video quality will be 
discussed in greater detail hereinafter. 

The processed DCT coefficients are produced at 
the output 108 of .DCT coefficients processor 107. 
These processed DCT coefficients, together with the 
motion vectors 106 produced at the second output of 
VMD 104 are sent to the video multiplexing encoder 
(VME) 109 to form a new video bit stream substantially 
conforming to the H.261 standard. The VME 109 en- 
codes the processed DCT coefficients and motion vec- 
tors, and multiplexes them into the layered data struc- 
tures shown in FIG. 3. to be described in greater detail 
hereinafter The new coded bit stream, produced at the 
output 110 of VME 109. is sent to a transmission buffer 
111. 

As it is well-known that compressed video data sig- 
nals may include a plurality of components, wherein 
each component may be represented by a different 
number of bits, transmission buffer 111 performs a var- 
iable-rate to constant-rate translation for the com- 
pressed video. The transmission buffer 111 includes cir- 
cuitry to ascertain and to indicate the status of the trans- 
mission buffer 111. which is defined as the occupancy 
ratio of the memory locations within the transmission 
buffer 111. The occupancy ratio refers to the ratio be- 
tween the number of occupied memory locations within 
a given buffer and the total number of memory locations 
within this buffer. The buffer status is produced at a first 
input/output 1 1 2 of the transmission buffer 1 1 1 This first 
output 112 is coupled to a rate control 113 circuit. This 
rate control 1 1 3 circuit adjusts the average data rate pro- 



vided by the DCT coefficients processor 107 

Rate control 1 1 3 circuit and DCT coefficients proc- 
essor 107 are coupled together via signal line 114. The 
rate control 1 1 3 circuit includes a first communications 

5 line 115 adapted for receiving a signal, (for example, 
from a control processor) which specifies a desired out- 
put bit rate for transmission buffer 111 The communi- 
cations line 115 is also adapted to transmit signals' to 
control processor 840 (FIG. 9) The desired output rate 

to signal is processed by rate control circuit n 3 in conjunc- 
tion with the buffer status signal received at the first in- 
put/output 112. to generate a rate control output signal 
which is downloaded via signal line 114 to DCT coeffi- 
cients processor 107. Based upon the desired output 

75 rate signal and the buffer status signal, the rate control 

1 1 3 circuitry computes the total number of bits for each 
frame, as well as the bits targeted for each macro block. 
The targeted bits per macro block or bits per frame are 
used as reference to produce a proper control signal. 

20 which is applied to a second input 11 4 of the DCT coef- 
ficients processor 107. The function of the second input 

114 is to force the DCT coefficients processor 107 to 
operate in such a manner that the bits produced for each 
macro block are as close to the targeted bits per macro 

2S block as possible. The type of signal applied to second 
input 114 is dependent upon the specific type of DCT 
coefficients processor 107 which is utilized. 

DCT coefficients processor 107 is the processing 
unit where the DCT coefficients recovered from the 

30 VMD 104 are further processed in order to match a de- 
sired (i.e. target) output video rate. Three methods can 
be used to process the DCT coefficients to reduce the 
total number of bits Each of these methods provides for 
the graceful degradation of video quality as was dis- 

o5 cussed above The first method is termed DCT coeffi- 
cients zeroing, the second method is called the requan- 
tization of the DCT coefficients, and the third method 
consists of the combination of the first and second meth- 
ods. In the first method. DCT coefficients are partitioned 

■to into groups based upon the relative importance of the 
various coefficents. Due to the fact that DCT coefficients 
are generally organized into two-dimensional arrays 
wherein the array entries which are relatively close to 
the upper left-hand corner of the array include relatively 
low-frequency components, as compared with array en- 
tries which are relatively close to the lower right-hand 
corner of the array, the relative importance of various 
DCT coefficients is known. The lower frequency com- 
ponents are more important and the higher frequency 

50 components are less important. Based upon the output 
produced by rate control 1 1 3 circuit on signal line 114. 
the coefficients of the least important group are set to 
zeroes. Here, the control signal on signal line 114 con- 
sists of a digital representation of the indices of a plu- 

55 rality of specific importance groups or simply indices of 
the DCT coefficients within a macro block, whose coef- 
ficients will subsequently be set to zeroes By forcing 
some DCT coefficients to zero the amount of data pro- 
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duced by the DCT coefficients processor 107 can be 
properly controlled by rate control 11 3 circuit. 

FIGs. 2 and 3 are data structure diagrams setting 
forth illustrative coding formats for representing video 
information in accordance with the H.251 standard. Re- 
ferring now to FIG 2. video information consists of a plu- 
rality of frames 201. 203. 205. 207. 209. 211. 21 3. 215, 
217 219. 221. Each of these frames contains a a rep- 
resentation of a two-dimensional video image in the 
form of a pixel array. Since a given frame may represent 
a video image at a specific moment in time, a plurality 
of frames may be employed to represent a moving im- 
age. Each frame represents a picture. A plurality of 
frames represents a coded video sequence. 

Each of the frames is compressed according to any 
one of two types of compression algorithms, termed in- 
tra-frame coding (I) and predictive coding tP). For ex- 
ample, frames 201 and 211 are compressed using intra- 
frame coding (I), and frames 203. 205. 207 209. 21 3. 
21 5. 217. 219. and 221 are compressed using predictive 
coding. The sequence of frames shown in FIG. 2 estab- 
lish a data structure for representing a video image in 
the form of an encoded video sequence having a plural- 
ity of levels arranged in a two-dimensional array, where- 
in each level represents the value of a pixel element. 
This encoded video sequence may be termed a coded 
video bit stream. 

If intra-frame coding *i) is to be applied to a given 
frame, sjch as frame 201 . the frame :s iermed an l-des- 
ignated ;ra~:e. and if predictive coding fp) is to be ap- 
plied to a given frame, such as frame 205. the frame -s 
termed a P-designated frame. 

Pursuant to intra-frame coding (I) compression 
processes, the l-designated frame 201 is divided into a 
plurality of pixel blocks, wherein each block consists of 
an array of 5 x 3 pixels. Next, a discrete cosine transform 
(hereinafter. DCT). is performed on each of the pixels in 
the 8x5 pixel block, in accordance with procedures well- 
known to those skilled in the art. to generate a plurality 
of DCT coefficients Thereafter, quantization is per- 
formed off the DCT coefficients, in accordance with well- 
known quantization procedures. These quantized DCT 
coefficients constitute compressed video image infor- 
mation for the l-encoded frame 201. 

Predictive coding (P) is implemented on a P-desig- 
nated frame, such as frame 205 by: 1) partitioning the 
P- designated frame into a plurality of macro blocks For 
example, if the frame includes a plurality of pixel arrays, 
each having 16 x 16 pixels (FIG 2. 251. 252.253.254. 
257. 258). each 16 x 16 pixel array may be partitioned 
into 4 contiguous blocks, wherein each block is an 8 x 
8 pixel array: a 16 x 16 pixel array (luminance) together 
with an = x 3 pixel block (chrominance) and an 5 x 9 
pixel block (chrominance), comprises a macro block 
247: 2) for each of the macro blocks (16 x 16 pixels) 
created in step ( 1 ). searching the most recent previously 
occurring frame (which could be either a P- or an l-des- 
ignaled frame but m the present example is frame 203) 



for the macro block which contains image information 
that is most similar to the image information in the macro 
block created in step (1), 3) generating motion vectors 
to spatially translate the macro block found in the prior 

5 I or P frame in step (2) to the location of the similar macro 
block in the P frame presently being compressed. 4) 
generating a predicted frame from the most recent pre- 
viously occurring frame using the motion vectors: 5) on 
a macro-btock-by-macro-block basis, subtracting the 

io predicted frame from the P-frame being compressed, to 
generate blocks of residues: 6) performing DCT's on the 
blocks of residues: 7) quantizing the coefficients of the 
blocks of transformed residues: and B) concatenating 
the quantized residue coefficients and the motion vec- 

'5 tors to form a compressed video signal. 

In an intra-frame coded (I) picture, every macro 
block is in::a-coded. That is. each macro block is coded 
without referring to any macro block in the previous 1-or 
P-frame. In the predictive-coded (P) picture, the macro 

20 block can be either intra-coded or inter-coded. 

To form the coded video bit stream for transmission, 
the compressed image information, as well as other in- 
formation such as motion vectors, are coded using 
specified code words. The code words are then multi- 

25 plexed into a layered data structure to form the final bit 
stream. In an H.26l-like environment, the coded bit 
stream is organized into a hierarchical format, the struc- 
ture of which is illustrated in FIG. 3. 

Referring to FIG 2. the sequence of frames 201. 

30 203. 205. 2C7 209. 211. 213. 215. 217. 219. 221 forms 
a coded video bit stream. This bit stream may be con- 
ceptualized as a serial representation of coded frames 
which can be processed to form a moving video image 
(i.e.. a moving picture). A typical sequence of frames is 

35 tPPR.PiPPP. where I indicates an intra-coded frame, 
and P designates a predictive-coded frame. For each 
frame 221 . the coded bit stream representing the frame 
includes a header 263 and coded data 265. Each head- 
er 263 includes a start code and data related to the re- 

-to spective frame (t.e. : picture). In an H.261 system envi- 
ronment, much of the header information is required for 
synchronization purposes. For example, at the frame 
(picture) layer for frame 221 . header 263 includes a pic- 
ture start code (PCS) field 267. a picture number (TR) 
field 269. a picture type (PTYPE) field 271. a PEI field 
273. and a PSPARE field 274. The PEI field 273 and the 
PSPARE field 274 are adapted to accommodate extra 
information which may be required for future applica- 
tions. 

50 Picture data is segmented into Groups of Blocks 
(GOB) 223. 225. 227. 229. 231 . 233. 235. 237. 239. 24 1 . 
243. and 245. A GOB (for example. GOB 229) compris- 
es one-twelfth of the coded frame (CIF) 221 Therefore. 
GOB 229 may be conceptualized as including one-third 

55 of one quarter of a coded picture area The area repre- 
sented by one-quarter of a coded picture ts abbreviated 
asQClF. Accordingly, there are 12 GOBs 223 225 227 
229. 231 233. 235. 237 239 241. 243. 245 tn a CIF 
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frame 221. and three GOBS in a QCIF frame. The ar- 
rangements of GOBs in a CI F/QCIF picture are depicted 
in FIGs. 2 and 3. 

Each GOB 229 includes a header field 291. fol- 
lowed by a macro block data field 293. The header field 5 
291 includes a GOB start code (GBSC) field 292. a 
group number (GN) field 293. a group type (GTYPE) 
field 294. a GOB quantizer (GGUANT) field 295. and 
spare information fields in the form of GEI field 296 and 
GSPARE field 297 Each GOB 229 consists of 33 macro io 
blocks, such as "macro block 24" (reference numeral 
247) and "macro block 25" (reference numeral 249). The 
arrangement of macro blocks within a GOB is depicted 
in FIG. 2. 

Each macro block includes a header field 275 fol- '5 
lowed by a block data field 277. The header field 275 
includes a macro block address (MBA) field 279. a block 
type information (MTYPE) field 281 . a quantizer type 
(MQUANT) field 283. a motion vector (MVD) field 285. ' 
and a coded block pattern (CBP) field 2S7 The block 20 
data field 277 of each macro biock 247 consists of 6 
blocks, including four luminance blocks Yi (reference 
numeral 251). Y2 (reference numeral 252). Y3 (refer- 
ence numeral 253). Y4 (reference numeral 254). one 
chrominance block U (reference numeral 257). and one 25 
chrominance block V (reference numeral 259). An illus- 
trative example of the contents of luminance block U 
(reference numeral 257) is set forth in FIG. 2. Note that 
this block includes an 6 x 3 pixel array wherein all pixels 
have a luminance value of black. 30 

A block represents a matrix (array) of pixels, e. g.. 
3x6. over which a discrete cosine transform (DCT) is 
performed. The array of pixels is represented by a matrix 
of pixel array coefficients, including a DC coefficient and 
a plurality of AC coefficients. The transformed coeffi- 35 
cients (TCOEFF) 301 (FIG. 3) consists of a DCT DC co- 
efficient occurring first, followed by respective pixel ar- 
ray coefficients (AC), in the order of their relative impor- 
tance. The arrangement of DC and AC coefficients in an 
illustrative block data field 277 (FIG. 3) is shown in FIG. -to 
4. The block data field 277 (FIG. 3) consists of the trans- 
formed coefficients (TCOEFF) 301 and an end of block 
code (EOB) 303 which is appended at the end of each 
successively occurring block of data. 

A typical partitioning of DCT coefficients is illustrat- 
ed in FIG. 5. The DCT coefficients are arranged in a two- 
dimensional array 500 stored in block data field 277 
(FIG. 3). The two-dimensional array 500 (FIG. 5) in- 
cludes eight rows and eight columns. Each entry in the 
array corresponds to a specific entry group, such as 50 
Group 506. Group 507. or Group 508. The groups are 
based upon the relative importance of the entries con- 
tained therein. Each group includes entries conforming 
to a specific range of importance levels. These impor- 
tance levels relate to the relative extent to which the 55 
elimination of a particular entry would degrade the qual- 
ity of the overall video image in a given frame. In the 
example of FIG 5 Group 506 is the most important 



group, and includes entries having a relatively high level 
of importance Group 507 includes entries having an in- 
termediate level of importance, and Group 508 includes 
entries having the least importance to the overall quality 
of the video image. 

FIG. 6 is a software flowchart setting forth a proce- 
dure for performing video bit rate matching according to 
a preferred embodiment disclosed herein. The opera- 
tions set forth in the flowchart may be implemented us- 
ing the hardware previously described in connection 
with FIG. 1. wherein the functioning of rate control 113 
circuit was disclosed. As shown in FIG. 6. the video bit 
rate matching procedure consists of six steps. The first 
step is initialization 600. followed by macro block 
processing 601. macro block parameler updating 602. 
and macro block counting 603. At block 603. a test is 
performed to ascertain whether or not the number of 
macro blocks that have already been processed are 
greater than a maximum number of macro blocks. If not. 
the program loops back to block 601 If so. the program 
progresses to block 604. where frame parameters are 
updated. Next, at block 605. frames are counted, and a 
test is performed to ascertain whether or not there are 
additional frames to consider. If so. the program loops 
back to block 601 . If not. the program ends. 

Referring to block 600. in the initialization stage, the 
rate control unit 1 1 3 (FIG. 1 ) performs the following se- 
ries of operations: 

1 . Obtaining a value for the desired video output bit 
rate. This value may be received for example, from 
one or more endpoint devices: 

2. Specifying the maximum output frame rate based 
upon the desired video output bit rate: 

3. Sending the maximum output frame rate to one 
or more endpoint devices: i.e.. sender endpoint 
devices--which are to be used to send video infor- 
mation to other endpoint devices. Once the sender 
endpoint device receives the maximum output 
frame rate, this endpoint device is forced to encode 
video signals with the specified maximum frame 
rate. 

4 Computing the average number of bits per frame 
and stores this number in a memory location des- 
ignated as average_bits_per Jrame. If the desired 
video output bit rate is R ou , and the maximum frame 
rate is F oul . then the average_bits_per_frame= 

R _out 
r cur 

5. Initializing the transmission buffer with an initial 
buffer fullness specified as B 0 . 

6. Specifying the targeted_bits_perjrame for the 

1st frame. targeted_bits_per_frame = K'-^i 

cut 

where K is a constant which is chosen based on the 
maximum frame rate and the initial buffer fullness 
B 0 . Then the largeted_bits_per_mblk is targeted. 
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bits per mblkr"^-^^;^ 

7. According to the targeted_bits_per_mblk. speci- 
fying the particular group indices which are to be set 
to zero. 

At the macro block processing step (FIG. 6. block 
601), the DCT coefficients processor 107 (FIG. 1) per- 
forms the following steps: 

1. Receiving a macro block from theVMD 104 (FIG. 

1): 

2. Receiving a control signal over signal line 114 
from rate control 113 circuit in the control signal 
specifies one or more DCT coefficients to be set to 
zero: and 

3 Setting the DCT coefficients specified by the con- 
trol signal to zero 

After processing one macro block, the rate control 
1 1 3 circuit receives the new buffer status from transmis- 
sion buffer 1 1 1 via first input/output 1 1 2. The rate control 
1 1 3 circuit uses the buffer status, defined above lo up- 
date the control signal. Since the transmission buffer 1 1 1 
is used as a temporary storage facility for video infor- 
mation, the buffer may operate in a first state, where the 
buffer is in the process of receiving new video informa- 
tion from input buffer 110. or the buffer may operate in 
a second state, where the buffer has already stored in- 
coming video information and is adapted to output this 
information to buffer output 116. Therefore, the status 
of the buffer refers to the state in which the buffer is op- 
erating at a given moment. 

The steps implemented by rate control circuit 113 
for updating the control signal include: 



1 Obtaining the transmission buffer 11 1 status. 
2. Obtaining the total number of bits used in the 
frame: 

3 Based on the targeted memory location occu- 
pancy rate for transmission buffer 111 (i.e.. buffer 
fullness), computing the targeted bits for the next 
frame and the targeted bits for each macro block: 

4 Based on the targeted bits for each macro block, 
providing an appropriate control signal for the 1st 
macro block of the next frame. 



After frame parameter updating, the new frame is 
checked. If there are no more bits, then the procedure 
ends. Otherwise, the procedure reverts back to process- 
's ing macro blocks. 

A second method of video bit rate matching is the 
requantization of DCT coefficients. The output signal at 
the first output 105 of VMD 104 includes two compo- 
nents: quantized DCT coefficients, and a quantization 
20 parameter In order to determine values for the DCT co- 
efficients; an inverse quantization operation is per- 
formed on the quantized DCT coefficients as follows. Let 
{X;. i=0. 1.2... 53[ be the quantized DCT coefficients and 
(y h i=0. 1... 63} be the reconstructed DCT coefficients. 
25 with Qp representing the quantization parameter. Then, 
with respect to an H. 261 -like environment, in the (-cod- 
ing mode, the reconstructed DC coefficient y 0 is calcu- 
lated using the relationship 

Vo = x o * 5 

^° and the remaining coefficients are calculated using the 
formula 

y ; = [Xj* 2 + sin fx.)] ' Op. 

where {i= 1.2.. .63} in I mode and {i=0. 1. . 63} in P 
35 mode, and the sign(w) function is defined as follows: 



1. Obtaining the total number of bits used for the 
mblk. bits_per_mblk: 

2. Computing the difference between the targeted^ 
bits_per_mblk and the actual bits_per_mbfk. 
bits_difference +r targeted_bits_per_mblk-bits_ 
per_mblk. 

3. Updating the control signal on signal line 114 
based on the following: 

If difference > 0 reduce the number of indices to 
be set to zero. 

else if difference < 0. increase the number of indices 
to be set to zero, 
else no change 

At the end of processing each macro block, the 
macro block counter is checked against the total number 
of mblk to ascertain whether or not a frame is finished. 
If a frame is finished, rate control 113 circuit starts up- 
dating the frame parameters. At block 604 the rate con- 
trol 113 circuit performs the following series of opera- 
tions. 



sien(w) = 



40 



I if w>0 
-/ if w<0 



To control the amount of data produced by the DCT 
coefficients processor 107 (FIG. 1)- the rate-control unit 
computes the proper quantization parameter Op new 
based on the targeted bits per macro block and sends 

^5 it to the DCT coefficients processor 107 lo requantize 
the DCT coefficients. Let \z r i=0.i. .63} be the new 
quantized DCT coefficients and Op new be the new 
quantization parameter obtained from the rate control 
113 circuit. Then, the new quantized DCT coefficients 

so are determined by 

z 0 = <yo +4 > /B - 

where z 0 is the DC coefficient of the i -coded macro 
block. The rest of the coefficients are obtained by 
z- = y.(2*Op ) 

where {i=l. 63 } for the mtra-coded macro block and 
(i=0.1... .63} for inter-coded macro blocks. 

With respect to the second method of matching vtd- 
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eo signal bit rates, the sequence of operations per- 
formed by the hardware configuration of FIG. 1 is virtu- 
ally identical to the process set forth in FIG. 6. with the 
following exceptions. Referring back to FIG. 6. at the in- 
itialization stage (block 600). the rate control 113 circuit 
performs the following steps' 

1 Obtaining the new (desired and/or target) video 
output bit rate: 

2. According to the new video output bit rate, spec- 
ifying the maximum output frame rate: 

3. Sending the maximum frame rate via communi- 
cations fine 115 to the sender endpoint device 
(defined above) to force this endpoint to encode the 
video with the maximum frame rate. 

4. Computing the average number of bits per frame 
as average _bits_per Jrame. Let new video bit rate 
be R out and the maximum frame rate be F OUI . then 

R 

the average_bits_per_frame=-^4. 

QUI 

5. Initializing the transmission buffer with an initial 
buffer memory occupancy rate (fullness) of B 0 . 

6. Specifying the targeted_bits_perjrame for the 
1st frame 

ft , 

targeted_bits_per_frame = K* ~^ 

out 

where K is a constant which is chosen based on the 
maximum frame rate and the initial buffer fullness 
B 0 . Then the targeted_bits_per_mblk is 

ta rgeted_ bi(s_ per_ frame 
total _number_ot_ mblk 

7. According to the targeted_bits_per_mblk, specify 
the new quantization parameter Qp new . 



targeted_bits_per_mbik : 



At the macro block processing step (block 601 ). the 
DCT coefficients processor 107 performs the following 
steps: 

1. Obtaining an mblk from the VMD 104: 

2. Performing inverse quantization based on the Qp 
and recovering the DCT coefficients: 

3. Obtaining the control signal on signal line 114 
from rate control 113 circuit: 

4. Using the control signal on signal line 114. 
requantizing the DCT coefficients. 

After the processing of one macro block has been 
completed, the rate control 11 3 circuit 113 obtains the 
new (current) transmission buffer 111 status and up- 
dates the control signal on signal line 114. The steps 
implemented by rate control 1 1 3 circuit include: 

1. Obtaining the total number of bits used for the 
macro block. bits_per_mblk: 

2. Computing the difference between the 
targeted_bits_per_mbtk and the actual bits_per_ 
mblk: 



is 
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45 
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bits_difference+=targeted_bits_per_mblk-bits_per_mblk. 

3. Updating the control signal 1 1 4 based on the fol- 
lowing: 

If difference > 0 reduce the size of the quanti- 
zation parameter: else if difference < 0. increase the 
size of the quantization parameter: else no change. 

At the end of processing each macro block, a macro 
block counter which counts the number of macro blocks 
which have been processed, is checked against the total 
number of macro blocks to ascertain whether or not a 
frame has been completed If a frame has been com- 
pleted, rate control 1 1 3 circuit commences updating the 
frame parameters. At block 604. the rate control 1 1 3 cir- 
cuit performs the following steps: 

1. Obtaining the transmission buffer 111 status: 

2. Obtaining the total bits used by the frame: 

3. Based upon the targeted buffer fullness (memory 
location occupancy rate), computing the targeted 
bits for the next frame and the targeted bits for each 
macro block: 

4. Based on the targeted bits for each macro block, 
generating an appropriate control signal for the first 
macro block of the next frame. 

After frame parameter updating, the new frame is 
checked. If there are no more bits, then the procedure 
ends. Otherwise, the procedure reverts back to the mac- 
ro block processing step at block 601 . 

A third method of video bit rate matching may be 
employed in conjunction with a preferred embodiment 
disclosed herein. This third method includes all methods 
which represent combinations of various features of the 
first and second methods. The manner in which the first 
and second methods are combined is determined by the 
specific applications of a given system. One illustrative 
combination of the first and second methods is the proc- 
ess of using DCT coefficient partitioning to process in- 
tra-coded macro blocks, and then employing requanti- 
zation to process the inter-coded macro blocks. 

Although system 1 00 (FIG. 1 ) with the DCT proces- 
sor equipped for implementing the three different 
processing schemes described above is satisfactory for 
lower rate reduction and intra-coded frames, there is a 
mismatch, "drift" between an endpoint device that trans- 
mits video information at a fast rate relative toother end- 
point devices which decode this video information at a 
slower rate. This mismatch is brought about because 
the video encoder is required to operate at a faster bit 
rate than the video decoder. This mismatch exists for all 
the inter-coded frames and is likely to accumulate with 
time, unless an intra-coded frame is periodically insert- 
ed into the video bit stream. To control the accumulation 
of the mismatch an improved DCT processor with the 
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mismatch correction elements is shown in FIG, 7 

FIG. 7 is a hardware block diagram setting forth an 
illustrative structure lor the discrete cosine transforma- 
tion (DCT) processor ot FIG. 1. The hardware configu- 
ration of FIG. 7 represents an improvement over the 5 
DCT coefficient processor disclosed before in connec- 
tion with FIG. 1 . as well as other existing stale of the art 
systems, such as the systems described in an ITU-T 
document entitled. "Low Bitrate Coding (LBC) for Video- 
phone" document no. LBC-94-166. This document de- io 
scribes methods for reducing the bit rate of compressed 
video information with a minimal amount of associated 
processing delays One implementation described in 
the ITU document utilizes one motion -compensated 
prediction storage device and two transform operations: '5 
a forward transform operation, and an inverse transform 
operation. The main purpose of this implementation is 
to correct the "drift", i e. the mismatch, between a video 
encoder and a video decoder. 

According to a preferred embodiment disclosed 20 
herein, the two transform operations described in the 
preceding paragraph are no longer required. Rather, 
motion compensation is performed in the transform do- 
main, as the terms "motion compensation" and "transfer 
domain" are generally understood by those skilled in the 25 
art. With reference to FIG. 7. one feature of this embod- 
iment is that the drift error signal stored in a picture mem- 
ory of a prediction frame storage device 703 need not 
be stored with fui: accuracy In particular, only a small 
number of the lower-frequency components of the trans- 30 
form coefficients need to be retained in the picture mem- 
ory. Since only a relatively small number of coefficients 
are now involved in the motion compensation process, 
and the transform operations are no longer needed, im- 
plementation of the embodiments disclosed herein is 
simplified considerably over the system described in the 
above-referenced ITU-T document identified as no. 
LBC-94-166. 

The simplified system disclosed herein is described 
below with reference to FIG. 7. An improved DCT (dis- JO 
crete cosine transformation) processor 107 is shown, 
which includes an inverse quantizer 701. a quantizer 
702. and a prediction frame storage device 703. The in- 
verse quantizer 701 accepts DCT coefficients and quan- 
tization parameters of an input bit stream from the first ^ 
output 105 of VMD 104 (FIG. 1) The output of inverse 
quantizer 701 which is the reconstructed DCT coeffi- 
cient, is coupled to a first input of a summer 704. and 
this output is also coupled to a first input of a subtractor 
706. The output of summer 704 is fed to a first input of so 
quantizer 702. A second input of quantizer 702 is under 
control of the signal line 114 which is coupled to rate 
control 113 circuit (FIG. 1), 

The output of quantizer 702 (FIG. 7) which is the re- 
quanttzed DCT coefficient 10S is fed to a second input 55 
of subtractor 706 The output of subtractor 706. which 
is the difference between output 707 and output 10c. 
representing the DCT coefficients of the error signals i 
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e "drift" signal is connected to a lirst input of summer 
705. The output of summer 705 is coupled to a first input 
of prediction frame memory storage device 703. and a 
second input of prediction frame memory storage device 
703 is connected to the second output of VMD 104 (FIG 
1 ). The output of prediction frame storage device 703 is 
fed to a second input of summer 704 and this output is 
also fed to a second input of summer 705. 

Inverse quantizer 701. quantizer 702. summers 
704. 705. and subtractor 706 are system components 
which are well-known to those skilled in the art. Conven- 
tional components may be used for these items With 
respect to the prediction frame storage device 703. this 
device includes a video buffer for storing information 
corresponding to one or more video frames, a random- 
access memory device, and a microprocessor for con- 
trolling the operation of the buffer and the random-ac- 
cess memory. The microprocessor is equipped to exe- 
cute a software program adapted to perform the steps 
outlined below in connection with the prediction frame 
storage device 703. 

The hardware configuration of FIG 7 operates as 
follows. Assume that the quantized DCT coefficients of 
an input video bit stream having a bit rate of R1 passes 
from the first output 105 of VMD 104 (FIG. 1) to the input 
of inverse quantizer 701 (FIG. 7). At inverse quantizer 
701. the quantified DCT coefficients are re-constructed 
to produce the DCT coefficients. The DCT coefficients 
plus the DCT coefficients of the "drift" error are sent to 
DCT coefficient processor 1 07 One purpose of DCT co- 
efficients processor 107 (FIGs. 1 and 7) is to generate 
an output signal representing the processed coeffi- 
cients. When the DCT coefficients processor 1 07 is in- 
itially started up. there is no drift error between output 
707 and output 109 (FIG. 7). Therefore, upon initial star- 
tup, inverse quantizer 701 provides an output signal in- 
cluding reconstructed DCT coefficients, and this signal 
passes unchanged through summer 704. to quantizer 
702. 

The operation of quantizer 702 is controlled by a 
signal on signal line 1 1 4 from the rate control 1 1 3 circuit 
(FIG. 1). so as to provide a signal having the desired 
output bit rate at the output of the bit rate matching sys- 
tem of FIG. 1 . The output of this bit rate matching system 
may be provided from the transmission buffer 111 (FIG. 
1 ). Note that the output of quantizer 702 (FIG. 7) repre- 
sents the DCT coefficients processor output 103 This 
output 10S is then recoded and multiplexed with motion 
vectors and quantization information by VME 109 (FIG. 
1). The VME 109 may then send the recoded. multi- 
plexed signal to transmission buffer 111. The signal is 
stored in the transmission buffer ill prior to transmis- 
sion at the desired output bit rate. The fullness, or buffer 
memory location occupancy ratio, of transmission buffer 
111 is used to control the quantization levels for quan- 
tizer 702 

Next, assume that the output of inverse quantizer 
701 does not equal the output of quantizer 702. The out- 
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put 707 of inverse quantizer 701 will be denoted as "A" 
and the output 1 08 of quantizer 702 will be denoted as 
"B". Thus, an error of B-A is added to the picture data. 
This error, denoted as Ed. is subtracted from the picture 
data by the system of FIG. 7. At initial startup. Ed is zero, 
and the data pass unchanged through summer 705 to 
the prediction frame storage device 703. Typically, only 
a small number of low-frequency coefficients are fed to 
subtractor 706. and thus. Ed is only an approximation 
of the actual drift error due to requantization During re- 
coding of the next video frame. Ed is approximately 
equal to the drift error of the previous frame. 

During motion-compensated prediction, prediction 
frame storage device 703 uses motion vectors on the 
second output 106 of VMD 104 (FIG. 1 ) to output a dis- 
placed drift error signal. As the prediction ts performed 
directly in the DCT domain, the output of the prediction 
frame storage device 703 representing the displaced 
drift error signal, is represented by its DCT coefficients. 

To compute the displaced drift error directly in the 
DCT domain, the following operations are performed by 
the prediction frame storage device 703. Since the mo- 
tion vector 106 represents an arbitrary number of pels, 
and the DCT representation of the reference frame 
stored in prediction frame storage device 703 are 
grouped into a block-based format, the motion compen- 
sated optimal block may overlay with four neighboring 
blocks in the DCT block structure. 

Let D r D 2 . D 3 . D 4 be the four neighboring (adjoin- 
ing) blocks in the reference frame, and PMC error be 
the DCT representation of the displaced drift error block. 
Then 
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where G h . G w are the DCT representations of the spatial 
sparse H n and H w matrices of the form. 
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where h and w represent overlay width. 

The G h2 and G w can be precomputed and stored in 
prediction frame storage device 703. As the drift error 
signal contains a lot of zeroes, the required computation 
is greatly reduced. Further, if motion vectors are zero, 
or integer multiples of the block width, the above block 
adjustment procedure can be avoided. This drift error 
signal wilt be seen at the DCT coefficients processor 
output 108. and at transmission buffer 111 (FIG. 1) which 
receives bits at the desired output bit rate Without cor- 
rection, this drift error will accumulate over time and 
eventually result in unacceptable system performance. 
In order to ameliorate the problem of drift error accumu- 
lation, the previous frame motion compensated drift er- 
ror Ed is added to the present frame signal (output 707) 
prior to requantization by quantizer 702 If quantizer 702 



introduced very little error, this would completely correct 
the drift error accumulation problem. However, since 
quantizer 702 introduces a finite amount of error, the drift 
can only be partially corrected, and the output of sub- 

5 tractor 706 will not. in general, be zero. Thus, summer 
705 adds the drift error from the current frame to the 
approximate accumulated drift error from previous 
frames to produce an approximate accumulated drift er- 
ror Ed for the current frame. 

w The prediction frame storage device 703 only has 
to compute a small number (i.e. . N) of compensated co- 
efficients. Note that, for intra-blocks of video data, the 
prediction frame storage device 703 is programmed to 
set Ed to zero. The relatively small number of computa- 

'5 tions required to implement the methods disclosed here- 
in is vastly reduced as contrasted with the relatively 
large number of computations required to perform ex- 
isting processes using pel domain motion compensa- 
tion An additional advantage of the disclosed methods 

20 is that these methods require much less memory space 
than existing prior art methods. 

The video transmission rate reduction system 
shown in FIG 1 can be implemented, for example, by 
using a general-purpose microprocessor a digital signal 

2$ processor (such as an AT&T DSP 3210 or an AT&T DSP 
1610). and/or a programmable video processing chip 
(such as an integrated circuit known to those skilled in 
the art as the ITT VCP chip). 

30 Multimedia System Using Video Processing of the 
Present Invention 

To illustrate various typical applications for the 
present invention in the context of multimedia confer- 
25 encing ; FIG. 5 shows a multimedia system using a video 
processor embodying the bit stream rate matching tech- 
niques disclosed herein. Referring nowto FIG. 3. a block 
diagram setting forth the system architecture of a mul- 
timedia conferencing system 800 is shown. The confer- 
va encing system includes an MCU 510. an ISDN network 
S04. and a plurality of endpoint devices such as first end- 
point device 301 . second endpoint device 302. and third 
endpoint device 803. 

Endpoint devices 801 . 802. and 303 are coupled to 
^5 MCU 810 via ISDN network S04. These endpoint devic- 
es 801 . 802. and 803 may include one or more user in- 
terface devices. Each interface device includes either 
an input means, an output means, or an input means 
combined with an output means Output means are 
50 adapted to convert multimedia electronic signals repre- 
senting audio, video, or data into actual audio, video, or 
data. Input means are adapted to accept audio, video, 
and/or data inputs, and to convert these inputs into elec- 
tronic signals representing audio video, and/or data. 
$5 Examples of user interface devices include video dis- 
play, keyboards, microphones, speakers, and video 
cameras, or the like. 

Endpoint devices 801 802. and 503 are adapted to 
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communicate using existing multimedia communication 
protocols such as ISDN The endpomt device multime- 
dia communication protocol controls the presentation of 
media streams (electronic signals representing audio, 
video, and/or data information) to the endpomt device 
user. Endpoint devices may function bi-directionally, 
both sending and receiving multimedia information, or 
alternatively, endpomt devices may function uni-direc- 
tional. receiving but not sending multimedia information, 
or sending but not receiving multimedia information. 

An example of a suitable endpoint device is an ITU- 
T H.320 audiovisual terminal, but any device capable of 
terminating a digital multimedia stream and presenting 
it to the user constitutes an endpoint device. A particular 
product example of an H 320-compatible endpoint is the 
AT&T-GIS Vistium 

MCU 210 is a computer-controlled device which in- 
cludes a multiplicity of communications ports, such as 
first communications port 870 and second communica- 
tions port 872. which may be selectively interconnected 
in a variety of ways to provide communication among a 
group of endpoint devices 801. 802. 803 Although the 
system of FiG. 8 shows two communications ports, this 
is done for illustrative purposes, as any convenient 
number of communications ports may be employed. 
MCU 8 10 also includes a control processor 840. an au- 
dio processor 841 a video processor 342. a data proc- 
essor 8^3. and a common internal switch 81 9. Each 
communications port includes a network interface, a de- 
multiplexer, and a multiplexer For example, firsi com- 
munications port 870 includes network interface 311. 
demultiplexer 81 3. and multiplexer 822. 

Although MCU 810 is shown with two communica- 
tions ports 870. 872 for purposes of illustration. MCU 
8 1 0 may. in fact include any convenient number of com- 
munications ports. For an MCU 31 0 having N ports, 
there are N network interfaces, one control processor, 
one audio processor, one video processor, and one data 
processor For each processor, there are N input signals 
coming from N demultiplexers and N output signals go- 
ing to the N multiplexers. Therefore. MCU 810 may be 
conceptualized as an N-port MCU where only two com- 
munications ports 870. 872 are explicitly shown. 

As shown in FIG. 8. first communications port 870 
includes network interface 811. demultiplexer 313. and 
multiplexer 812. Network interface 311 is a circuit which 
provides the conversion function between the standard 
line signal coding used by ISDN network 804 and the 
Px64 kbps H 221 signal used by MCU 810 Network in- 
terface 8ll includes output port 812. which is adapted 
to provide an output in the form of an H.221 signal. The 
H 221 signal is actually a multiplex of several different 
types of information ( audio, video data, control) there- 
fore, network interface 811 must send the incoming 
MCU H 221 signal to a demultiplexing device such as 
demultiplexer 813 Likewise network interface 611 has 
an input port 823 adapted to receive an H 221 signal 
from multiplexer 822 Multiplexer 822 combines a. plu- 



rality of individually-processed signals which are to be 
transmitted to a particular endpoint device 

Demultiplexer 31 3 separates an incoming multime- 
dia signal stream received from network interface 811 

5 into four components: a first component 3 1 4. comprising 
electronic signals representing control: a second com- 
ponent 615. comprising electronic signals representing 
audio, a third component 816. comprising electronic sig- 
nals representing video: and a fourth component 8i7. 

*o representing data. The first, second, third, and fourth 
components 8 1 4. 8 1 5. 3 1 6. 3 1 7 represent outputs of de- 
multiplexer 8 1 3 which are coupled to common internal 
switch 319 

Multiplexer 822 accepts a plurality of incoming mul- 

'5 timedia signal components from common internal 
switch 819. such as a first component 313 representing 
control, a second component 891 representing audio, a 
third component 620 representing video, and a fourth 
component 821 representing data. The multiplexer 522 

20 integrates the first, second, third, and fourth compo- 
nents 316.891. 820. 821 onto a single multimedia signal 
stream which is coupled to network interface 811. This 
single multimedia signal stream may be conceptualized 
as the output of multiplexer 822. The network interface 

25 311 routes this multimedia signal stream to a specific 
endpoint device 501 . 802. 803. For second communica- 
tions port 872. the four output components are first com- 
ponent 824 representing control, second component 
825. representing audio, third component 826. repre- 

30 senting video, and fourth component 827. representing 
data. The four input components to multiplexer 834 are 
first component 325. representing control, second com- 
ponent 829. representing audio, third component 330. 
representing video, and fourth component 831. repre- 

35 senting data. 

Common internal switch 31 9 contains a plurality of 
electronic switches, buffers, and/or amplifiers under the 
control of control processor 640. Common internal 
switch 61 9 is coupled to audio processor 841 for mixing 

J 0 and switching electronic signals representing audio: 
common internal switch 619 is also coupled to video 
processor 842 and data processor 843 for mixing and 
switching electronic signals representing video and da- 
ta, respectively. Therefore, common internal switch 619 

J 5 effectively receives four output components Irom each 
communications port 570. 572 and routes these output 
components to selected ones of respective processors 
(control processor 840. audio processor 641 video 
processor 842 and/or data processor 843) within MCU 

50 310. Likewise, common internal switch 5 1 9 receives the 
output components of each processor in MCU 310 and 
routes these outputs to the multiplexer 822 of each com- 
munications port 370. 

Common internal switch 8l9 receives output con- 

55 trol signals from control processor 840 over signal line 
651. and provides input control signals to control proc- 
essor 840 over signal line 850 Common internal switch 
819 receives output audio signals from audio processor 
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641 over signal line S53. and provides input audio sig- 
nals to audio processor 841 over signal line 352. Com- 
mon internal switch 31 9 receives output video signals 
from video processor 842 over signal line 355. and pro- 
vides input video signals to video processor 342 over 
signal line 854. Common internal switch 319 receives 
output data signals from data processor 843 over signal 
line 857. and provides input data signals to data proc- 
essor 843 over signal line 856. Control processor 840 
provides control signals to the audio processor 841 . vid- 
eo processor 842. and data processor 843 over signal 
line 844. 

ISDN network 804 is connected to MCU 810 over 
signal line 805. Within MCU 810. signal line 805 is par- 
allel-connected to first and second communications 
ports 870. 372. For example, in the case of first commu- 
nications port 870. signal line 805 is connected to net- 
work interface 811. Network interface 811 is coupled to 
demultiplexer 31 3 over signal line 81 2. and this network 
interface 81 1 is also coupled to multiplexer 822 over sig- 
nal line 323. Signal line 512 is coupled to the input ter- 
minal of demultiplexer 313. and signal line 323 is cou- 
pled to the output terminal of multiplexer 822 

Audio processor 841 includes software and hard- 
ware for processing audio signals. The processing may 
take the form of switching the audio, mixing the audio, 
or both. In the case of audio mixing, the input signal to 
audio processor 841 is an aggregate audio signal con- 
sisting of each of the audio output signals from all of the 
communications ports 870. 872 of MCU 810. For an N- 
port MCU 810. this signal includes the N audio signals 
from the demultiplexers within each communications 
port 370. 872. 

To mix the audio, audio processor 841 decodes 
each of the audio inputs, linearly adds the signals ob- 
tained by decoding, andthen re-encodes the linearsum. 
For each endpoint device, this linear sum may be sub- 
jected to additional processing steps, so as to provide 
each endpoint device with audio information specific to 
that endpoint device. These additional processing steps 
may include, for example, any of the following: the out- 
put sum for a given endpoint device may exclude that 
endpoint's input: the sum may include inputs whose 
present or recent past values exceed a certain thresh- 
old: or the sum may be controlled from a specially-des- 
ignated endpoint device used by a person termed the 
"chair", thereby providing a feature generally known as 
chair-control. Therefore, the output of the audio proces- 
sor 641 is in the form ofN processed audio signals. 

In the case of audio switching, the input signal to 
audio processor 641 is a single audio signal which is 
selected from a given communications port 370 or 872. 
based upon control signals received from control proc- 
essor 840. No audio processing is implemented in the 
present example which involves only audio switching. 
The audio input is broadcast to all other audio processor 
341 outputs, either automatically or under manual con- 
trol. 
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Data processor 643 includes hardware and soft- 
ware means for implementing one or both of the func- 
tions generally known to those skilled in the art as 
"broadcast" or "MLP". For each type of broadcast data. 

$ data input is accepted from only one endpoint device at 
any one time. Therefore, the input signal to data proc- 
essor 843 is the data output from one of the communi- 
cations ports 870. 872. This data output is broadcast to 
the other endpoint devices as determined by control 

to processor 340. according to the capabilities of specific 
endpoint devices to receive such data, as set forth in the 
capability codes stored in memory units (RAM or ROM) 
of respective endpoint devices. 

Control processor 840 is responsible for determin- 
es ing the correct routing, mixing, switching, format and 
timing of the audio, video, data and control signals 
throughout a multimedia conference. The control proc- 
essor 840 retrieves one or more capability codes from 
each endpoint device Capability codes. which : are 

20 stored in endpoint device RAM and/or ROM. specify the 
audio, video, data, and/or control capabilities for this 
endpoint device. Control processor 640 retrieves the ca- 
pability codes from all N endpoint devices participating 
in a multimedia conference. These capability codes are 

25 stored in a memory unit (RAM) of MCU 610 so that con- 
trol processor 640 can correctly manage the conference 
for all endpoint devices. This storage may occur for ex- 
ample, in a random-access memory (RAM) device as- 
sociated with control processor 840. In turn. MCU 810 

30 sends the capability codes to each of the N communi- 
cations ports 670. 872 so that each of the endpoint de- 
vices 601. 802. 603 are enabled to communicate with 
MCU 310 at a bit rate determined by MCU 610 and ap- 
propriate for that specific endpoint device 301 . 302. 603. 

55 To properly control the operations in the video process- 
ing unit 842. MCU 610 sends back different capability 
codes for different video operations. If the endpoints are 
coupled to MCU 810 via communications links with dif- 
ferent transmission rates for switching operation, the 

•to minimum transfer rate is adapted in the capability code. 
The new code is sent to all the endpoint devices partic- 
ipating in the conference to force the endpoint devices 
to operate with the minimum transfer rate. For perform- 
ing a transmission rate matching operation, instead of 

^5 adapting the minimum transfer rate, a new maximum 
frame rate is specified in the capability code based on 
all the different transfer rates. The new capability codes, 
which include the specified maximum frame rates, are 
sent back to all the endpoint devices. In this case, the 

50 video bitstreams generated by the endpoint devices will 
have different transfer rates but the same frame rate. 

Control processor 640 receives inputs which are 
entered by conference participants into the user inter- 
face of an endpoint device 601 602. 503. These inputs 

ss are in the form of chair-control commands and com- 
mands embedded in bit streams conforming to the H. 
221 standard Commands from endpoint devices are 
routed to the control processor 640 to ensure the correct 
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distribution of bit streams to the audio, video, and data 
processors 641 842. 843 respectively to ensure that 
the correct audio decoding algorithm is used at the in- 
puts to an audio mixer within audio processor 841 . and 
to ensure that any incoming data is sent to a data broad- 
cast unit or MLP processor within data processor 843 

The control processor 840 also directs the switching - 
of the bit streams Irom the audio, video, and data proc- 
essors 841. B42. 843. respectively to each multiplexer 
822. 834. and specifies the audio encoding algorithm 
used in the audio mixer of audio processor 64 1 . and the 
algorithm used at each output from the audio mixer. The 
bit streams are routed to and from the various proces- 
sors 841. 842. =43 by the common internal switch 819. 
which is under control of the control processor 840. 

Video processor 842 processes the video signals 
received from the common internal switch =19. The 
processing may take the form of switching the video, or 
matching the video bit rate. In video switching, the video 
processor 842 receives one selected video signal from 
the switch 819 and transmits the video signal to some 
or all other endpomt devices participating in a given mul- 
timedia conference. Video selection may be automatic 
or under manual control. For instance the audio proc- 
essor =41 and the video processor 842 may be auto- 
matically controlled by control processor =40. such that 
an endpomt device with currently active audio (i.e. an 
endpoint device used by the "present speaker" which 
provides an audio signal to MCU 8 10 above a predeter- 
mined audio amplitude threshold) receives the picture 
of the endpoint device which previously had active audio 
(i.e.. an endpomt device used by the "previous speak- 
er"), while all other endpoint devices receive the picture 
of the present speaker 

A lime delay may be incorporated into the video 
switching implemented by video processor 842 to avoid 
excessively frequent video image changes caused by 
spurious sounds. As in the case of audio switching, vid- 
eo switching may be controlled directly from a specially- 
designated endpoint device used by a person termed 
the "chair". If the delay in the video processor 642 and 
the delay in the audio processor 641 differ by a signifi- 
cant (humanly perceptible) amount, a compensating de- 
lay may be inserted into the appropriate bit stream to 
retain lip synchronization. 

To match video bit rates, video processor 642 incor- 
porates the video transmission rate matching tech- 
niques of the present invention. 

With reference to FIG 9. the hardware conligura- 
tion of an illustrative stand-alone video processor 642 is 
shown. This video processor 642 may be employed with 
the MCU 610 of FIG. 6 or. alternatively, the video proc- 
essor =42 may exist as a stand-alone unit. Such a stand- 
alone unit may be used to match bit rates for all types 
of digital video information The video processor =42 
consists of a video transmission rate reduction una 100. 
a bit stuffing unit 200. an input switcher 905. and an out- 
put switcher 906 if desired the input switcher 905 and 



the output switcher 906 could be combined into a single 
integrated switcher unit, such as common interna! 
switch 319 (FIG. 8) and. hence would not be incorpo- 
rated into video processor 642. The inputs to the video 

5 processor 642. are for an N-port MCU. an the N-coded 
video bit stream obtained from N demultiplexers. The 
outputs of the system are the N-coded video bit streams 
which are processed to have different transfer rates. 
The outputs are the inputs to the N multiplexers The 

w number of required transmission rate reduction units 
and bit stuffing units is proportional to the difference in 
the transmission rates among the various endpoint de- 
vices. If there are M different video transfer rates in- 
volved in a conference, the video processor 842 needs 

'5 to'incorporate (M-1) bit rate reduction systems and *M- 
1) bit rate increasing units. In the configuration of FIG. 
9. M^2 for illustrative purposes. 

Signals produced by video processor 842 are cou- 
pled to signal line 644 which is used to convey the con- 

20 trof signals which control input switcher 905. output 
switcher 906. transmission rate reduction unit 100 and 
bit stuffing unit 200. Input switcher 905 provides N-to-1 . 
N-to-2. and N-to-3 switching capabilities. Similarly, out- 
put switcher 906 provides 1-to-N. 2-to-N. and 3-to-N 

25 switching capabilities. The switching is controlled by 
control signals on signal line 644. These control signals 
are generated by the control processor 840. Control 
processor 640 has control capabilities for handling dif- 
ferent transmission rates among a plurality of endpoint 

30 devices. 

If the transmission rates among al! the endpoint de- 
vices are the same, the control signals sent out by con- 
trol processor 640 over signal line 644 serves to place 
video processor 842 into a switching mode. In such a 

35 case, video processor 642 works as a buffer and switch- 
er, such that the input switcher provides an N-to-1 switch 
and the output provides an l-to-(N-l) switch. A direct 
signal path between input switcher 905 and output 
switcher 906 is utilized, whereas video transmission rate 

■fO reduction unit 1 00 and bit stuffing unit 200 are not used. 
If the transmission rates among the endpoint devic- 
es are different, and if there are more than two different 
transmission rates among the endpoints. extra rate re- 
duction units 100 and extra bit stuffing units 200 are 
needed in the video processor 642. As an example. FIG. 
10 shows a video transmission bit rate matching system 
which uses live endpoint devices connected to an ISDN 
network 604 via a five-point connection wherein the 
switching functions of video processor 642 are mtegrat- 

50 ed into common internal switch 6 1 9. The ISDN network 
604 is connected to a five-port MCU 510. including first 
communications port 840 and additional communica- 
tions ports 37 1 . A first endpoint device endpomt A 901 . 
is connected to the ISDN network 604 via a 364 kbits/s 

55 communications link A second endpoint device end- 
point B 902. is connected via a 256 kbits/ s link. Third 
and fourth endpoint devices endpoint C 903 and end- 
point D 904 respectively are each connected to ISDN 
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network 304 via 125 kbits/s links, and a fifth endpomt 
device, endpoint E 905. is connected via a 64 kbtts/s 
link. Therefore, the MCU 910 must utilize a video proc- 
essor 842 having five inputs and five outputs. To pro- 
duce five outputs with three different transfer rates, the 
video processor needs three rate reduction units 920. 
921 . 922. three bit stuffing units 923. 924. 925. and an 
optional video buffer. The three rate reduction units 920. 
921. 922 provide respective rate reductions of 346-to- 
255 kbits/sec. 364-to-129 kbits/sec. and 384-to-64 
kbits/sec. The three bit stuffing units 923. 924. 925 pro- 
vide respective bit stuffing conversions of 64-to-l28 
kbits/sec. 64-to-255 kbits/sec. and 64-to-384 kbits/sec. 

To provide the proper switching for the inputs and 
the outputs, the control processor 841 has to provide 
proper control signals. If endpoint A 901 ts used by the 
current conference speaker and endpoint B 902 is used 
by the conference speaker who spoke immediately prior 
to the speaker using endpoint A 901. then the input 
switcher 305 provides an input switching function 
wherein 5 inputs are switched to 4 outputs, such that the 
coded bit stream from endpoint A 901 is switched to any 
one of the rate reduction units 920. 921. 922 which pro- 
vide respective bit rate reductions of 364-to-256 
kbits/sec. 384-to-l 28 kbits/sec. and 384-to-64 kbits/sec. 
To send the picture of the previous speaker to endpoint 
A 901 . the coded bit stream from endpoint B 902 is 
switched to bit stuffing unit 925. which performs a bit 
stuffing from a rate of 255 kbits/sec to a rate of 384 
kbits/sec. Via the output switcher 306 (FIG. 8). the out- 
put of rate reduction unit 920. which performs a bit rate 
reduction of 384 kbits/sec to 256 kbits/sec. is routed to 
endpoint B 902. the output of rate reduction unit 921. 
which performs a rate reduction of 364 kbits/sec to 1 23 
kbits/sec. is routed to endpoints C and D. 903. 904 re- 
spectively, and the output of rate reduction unit 922. per- 
forming a reduction of 334-to-64 kbits/sec. is routed to 
endpoint E 905. The output of bit stuffing unit 925 (64-to- 
384 kbits/sec), is routed to endpoint A 901. 

Claims 

1 . A bit stream rate matching apparatus characterized 
by: 

(a) bit rate increasing means for converting a 
video bit stream having a first transmission rate 
to a video bit stream having a second transmis- 
sion rate, wherein the first transmission rate is 
less than the second transmission rate: 

(b) bit rate decreasing means for converting a 
video bit stream having a third transmission 
rate to a video bit stream having a fourth trans- 
mission rate wherein the third transmission rate 
is greater than the fourth transmission rate 

ic) first switching means for switching each of 
a plurality of incoming video bit streams to any 



one of the bit rate increasing means the bit rate 
decreasing means and a second switching 
means: and 

(d) second switching means for switching video 
5 bit streams from the bit rate increasing means. 

the bit rate decreasing means, and the first 
switching device to each of a plurality of outgo- 
ing video bit streams. 

io 2. A bit stream rate matching apparatus as set forth in 
Claim 1 characterized in that each of the plurality of 
incoming video bit streams is from a corresponding 
endpoint device, and each of the plurality of outgo- 
ing video bit streams is directed to a corresponding 

>5 . endpoint device, specific ones of the incoming 
video bit streams being associated with specific 
ones of the outgoing video bit streams to form an 
associated incoming-outgoing video bit stream, 
each associated incoming-outgoing video bit 

20 stream corresponding to a particular endpoint 
device. 

3. A bit stream rate matching apparatus as set forth in 
Claim 1 further comprising processor means 540. 
25 coupled to the first and second switching means : bit 
rate increasing means, and bit rate reduction 
means, for controlling the first and second switching 
means. 

^o 4. A bit stream rate matching apparatus as set forth in 
Claim 2 further characterized by processor means, 
coupled to the first and second switching means for 
controlling the first and second switching means. 

35 5. A bit stream rate matching device characterized by 

(a) bit rate increasing means for converting a 
video bit stream having a first transmission rate 
to a video bit stream having a second transmis- 

40 sion rate, wherein the first transmission rate is 

less than the second transmission rate, and: 

(b) bit rate reducing means for converting a 
video bit stream having the second transmis- 
sion rate to a video bit stream having the first 

4 $ transmission rate. 

6. A video bit stream rate matching device for use in 
a system characterized by 

50 a first endpoint device characterized by video 

transceiver means for transmitting and receiv- 
ing a video bit stream having a first transmis- 
sion rate, and 

a second endpoint device characterized by 
55 video transceiver means for transmitting and 

receiving a video bit stream having a second 
transmission rate, wherein the second trans- 
mission rate differs from the first transmission 
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rate. 

the video bit stream rate matching device being 
coupled to the first and second endpotnt 
devices; 

5 

CHARACTERIZED IN THAT 

the video bit stream rate matching device 
includes rate conversion means for converting a 
video bit stream having a first transmission rate to 
a video bit stream having a second transmission io 
rate, and for converting a video bit stream having a 
second transmission rate to a video bit stream hav- 
ing a first transmission rate 

The video bit stream rate matching device of Claim '5 
1. further characterized by: 

(a) video multiplex decoder means for decoding 
and demultiplexing an input video bit stream 
having a first transmission rate: the video mui- ?o 
tiplex decoder having a first output adapted to 
generate quantized information in ihe form of 
quantized discretel cosine transformation coef- 
ficients in response to the input video bit 
stream, ihe video multiplex decoder means 25 
having a second output adapted to generate 
motion vectors in response to the input video 

bit stream: 

(b) video multiplex encoder means for encoding 
and multiplexing quantized discrete cosine so 
transformation coefficients with motion vectors 

to generate an output video bit stream; the 
video multiplex encoder means having a first 
input adapted to receive quantized direct 
cosine transformation coefficients, a second 35 
input adapted to receive motion vectors, and an 
output adapted to generate the output video bit 
stream; 

(c) discrete cosine transformation coefficients 
processor means for processing a plurality of 4 0 
quantized discrete cosine transformation coef- 
ficients generated from a decoded, demulti- 
plexed video bit stream having N bits, the proc- 
essor processing the quantized discrete cosine 
transformation coefficient such that, after ■*$ 
processing, the quantized discrete cosine 
transformation coefficients provide an 
encoded multiplexed video bit stream having 

M bits wherein M=N. the discrete cosine trans- 
formation coefficients processor means having so 
a rate control input for specifying a desired bit 
rate for the encoded, multiplexed video bit 
stream, and 

(d) rate control means for controlling the bit rate 

of the encoded multiplexed video bit stream, ss 
the rate control means having 

(i) a first communications port for receiving 



a desired bit rate, and adapted for coupling 
to an endpomt device: 

(ii) a second communications port for 
receiving a video bit stream having a trans- 
mission rate R: and 

(iii) an output port for generating a video bit 
stream transmission rate S 1 in response to 
the receipt of transmission rates at the first 
and second communications ports: 

characterized in that the first output of Ihe 
video multiplex decoder means is coupled to the 
direct cosine transformation coefficients processor 
means, the second output of the video multiplex 
. decoder means is coupled to the second input of 
the video multiplex encoder means, the output of 
the video multiplex encoder means is coupled to the 
second communications port of the rate control 
means, and the output port of the rate control 
means is coupled to the rate control input of the 
direct cosine transformation coefficients processor 

8. A digital video transmission rate matching system 
characterized by: 

a first switching means having N input ports, 
first, second, and third output ports, and a con- 
trol signal input port: the control signal port 
■ being adapted to receive a plurality of control 
signals, each control signal specifying a partic- 
ular set of connections between the N input 
ports and any of the first, second and third out- 
put ports: the first switching means, in response 
to the receipt of any one of the plurality of con- 
trol signals, connecting the N input ports to any 
of the first, second, and third output ports as 
specified by the any one of the plurality of con- 
trol signals: 

a second switching means having first, second, 
and third input ports. N output ports and a con- 
trol signal input port: the control signal input 
port being adapted to receive a plurality of con- 
trol signals, each control signal specifying a 
particular set of connections between any of the 
first, second, and third input ports and the N 
output ports: the second switching means, in 
response to the receipt of any one of the plu- 
rality of control signals, connecting the N output 
ports to any of the first, second and third input 
ports as specified by the any one of the plurality 
of control signals: 

a transmission rate reduction means having an 
input adapted lo receive a video bit stream of 
bit rate R bits per second, an output adapted to 
transmit a video bit stream of bit rate S 1 bits per 
second, wherein R is greater than S 1 and video 
bit stream compression means for transforming 
a video bit stream of bit rate R bits per second 
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to a video bit stream of bit rate S bits per sec- 
ond: 

a transmission rate increasing means having 
an input adapted to receive a video bit stream 
of bit rate S bits per second, an output adapted 5 
to transmit a video bit stream of bit rate R bits 
per second, wherein R is greater than S. and 
video bit stream expansion means for trans- 
forming a video bit stream of bit rate S bits per 
second to a video bit stream of bit rate R bits io 
per second: 

characterized in that the first output port of the 
first switching means is coupled to the input of 
the transmission rate reduction means, the sec- 
ond output port of the first switching means is '5 
coupled to the input of the transmission rate 
increasing means, the third output port of the 
first switching means is coupled to the third 
input port of the second switching means: 
the first input port of the second switching 20 
means is coupled to the output of the transmis- 
sion rate reduction means: the second input 
port of the second switching means is coupled 
to the output of the transmission rate increasing 
means: 25 
the N input ports of the first switching means 
are each adapted to receive an incoming video 
bit stream and the N output ports of the second 
switching means are each adapted to transmit 
an outgoing video bit stream. 30 

9. The digital video transmission rate matching sys- 
tem of Claim 4 characterized in that the transmis- 
sion rate reduction means is the video bit stream 
rate matching device of Claim 1 . 2 or 3. 35 

10. The digital video transmission rate matching sys- 
tem of Claim 4. 5. 6 or 7 characterized in that the 
transmission rate increasing means is a bit stuffing 
device. 40 

11. A method of providing video bit stream rate match- 
ing, characterized by the following steps: 

a) demultiplexing an incoming video bit stream -*5 
to generate a plurality of demultiplexed video 

bit streams: 

b) decoding each of a plurality of the demulti- 
plexed video bit streams: 

c) performing a discrete cosine transformation 50 
(OCT) on each of the demultiplexed, decoded 
video bit streams to generate a plurality of DCT 
coefficients: 

d) discarding a subset of the DCT coefficients: 

e) encoding the non-discarded DCT coeffi- 55 
cients to generate a plurality of encoded video 

bit streams: and 

f ) multiplexing the plurality of encoded video bit 



streams to generate a multiplexed encoded, 
outgoing video bit stream. 

12. A method of providing video bit stream rate match- 
ing as set forth in Claim 11. characterized in that the 
DCT coefficients include a first subset of DCT coef- 
ficients corresponding to low-frequency video infor- 
mation and a second subset of DCT coefficients 
corresponding to high-frequency video information, 
the step of discarding being performed by discard- 
ing the second subset of DCT coefficients. 

13. A method of providing video bit stream rate match- 
ing as set forth in Claim 12 characterized in that the 

.high-frequency video information corresponds to 
video information above approximately 4.5 mHz. 
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